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Copyright ® 2003 Stratford Technology Ltd 

This Undercover Report may be freely shared, copied;, and distributed among software 
enthusiasts provided that all of the following conditions are observed: 

1 . No alterations of any kind are made to the report; the document must be provided 
in its entirety and exactly as is, including this legal notice 

2. The report may not be published, either electronically or in printed form, 
anywhere other than the web site of the author (see below) nor presented in any 
formal setting unless explicitly authorized in writing and in advance by the 
copyright owner. 

3. The report may not be sold, nor used in connection with any training, consultancy, 
or other commercial purpose unless explicitly authorized in writing and in 
advance by the copyright owner. 

All other rights are reserved. The intellectual property rights of the owner have been 
asserted and will be defended. 



Undercover Reports are based on actual tests run on different application server products, 
and significant care has been taken to ensure that a consistent approach is used that will 
reveal meaningful results. However, as the reports are provided free of charge, no 
warranty is made nor any guarantees made as to the accuracy of the report. 



For updates, additions, and corrections, please see: 

www.scottcrawfordxom/undercover 



JBoss and JBoss Group arc a registered trademark and servicemark of Marc Fleury under 
operation by The JBoss Group, LLC. 

Java, J2EE, Enterprise JavaBeans, and EJB are trademarks of Sun Microsystems, Inc. in 
the US and other countries. 

All other trademarks referenced are those of their respective owners. 

Stratford Technology Ltd is not affiliated with The JBoss Group, LLC, or Sun 
Microsystems, Inc., nor does it have any commercial relationships with any J2EE vendor. 
The Undercover Report series is not endorsed by any J2EE vendor. 
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I. Introduction 



A. Concepts 

Undercover Reports are based upon investigation of how Enterprise JavaBean (EJB) 
application servers manage their EJB components. The data presented are derived from 
tests run using Undercover EJB Components, that is, components which are written 
specifically to monitor how they are being managed. When these components are 
deployed into a J2EE-compatiblc application server product, and placed under load by a 
test client, they reveal information about the workings of the product Issues such as 
instance pooling, passivation, and caching of EJB components can be explored using this 
technique. 

Undercover components are intended to answer simple questions of bean-management 
strategy. Typical of the questions that they hope to answer is "Does application server X 
pool the instances of session beans it uses to respond to clients, or does it create a new 
bean for each client?" They may further help with broad conclusions about whether the 
number of beans used will be affected by the degree of processing performed by a bean 
or the size of its client load. 

However, it is a mistake to read the data with more precision than they are intended to 
provide. A test that results in 5 bean instances may result in 45 bean instances when 
repeated. Those looking to draw precise scientific measurements will be disappointed for 
several reasons. First, timing factors in the tests are non-deterministic and will often vary 
the exact results. Second, the undercover components are themselves artificial as they 
are fewer, smaller, and more eccentric than the beans in a real J2EE application. Finally, 
the hardware on which the tests are run - an ordinary laptop computer - does not resemble 
that which would be used for a production J2EE system. 

A proper introduction to the technique is given in the separate document, Introduction to 
Undercover Components, which is available from www^scottcrawford. co m/undercover. 
A few of the most important points are repeated briefly here as well. 

This technique is not useful for performance benchmarking. The products tested use 
different virtual machines, different wire protocols, different memory configuration, and 
all have their own unique characteristics. No attempt has been made to provide a "level 
playing field". Therefore it is neither fair nor uscftil to compare the performance of one 
product to the performance of another product on the same test. It is sometimes useful to 
see how the same product will perform differently in response to two similar tests; these 
are the only performance comparisons that appear in Undercover Reports. 

Default configuration hits been used unless otherwise specified on a specific result. All 
of the products are configurable, and the management of EJBs will be strongly affected 
by configuration and tuning. Therefore these reports should never be taken as evidence 
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that a product can not behave in a desired fashion; rather, they reveal "out of the box*' 
behaviour. 

Only one specific version of each product is tested for each J2EE release; the target 
version for the tests is the first one identified by its vendor as being folly production 
capable. Since some vendors come to market with a full production release much more 
rapidly than others, this means that products that came out at very different limes can be 
compared in these reports. It is also important to remember that any problematic 
behaviours identified in the reports may have been improved by a patch or minor release 
which occurs very shortly after the first production release. 

These reports focus on tests based on RMI calls and JMS messages sent remotely to 
Enterprise JavaBeans. This narrow focus allows for a deep analysis but it also avoids 
areas, such as incoming HTTP requests, in which products are likely to have strengths. 



Bs Details of Tested Product and Platform 


Vendor: 


An open-source community centred around The JBoss Group 


Product: 


JRoss Application Server 


Version: 


3.0.0 


Certification: 


Not officially certified. JBoss claims EJB 2.0 and J2EE 1.3. 


Released: 


Summer 2002 


URL: 


http://www.j boss.org/ 


Platform: 


Windows 2000 Professional 


Hardware: 


IBM ThinkPad T20 with Pentium til and 384 MB RAM 
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C. Document History 



Date 


J2EE 
version 


JBoss 
version 


Document 
version 


Comments 


4 June 2003 


1.3 


3.0 


1,0 


First published version 













Please Note: 

The results presented on the following pages are accompanied by only a brief explanation 
giving the purpose of each test. This avoids repetition of background material in all of 
the product-specific reports. For details on the mechanics of how the testing works, 
please refer to the separate document Introduction to Undercover Components. To 
ensure sensible interpretation of the results, it is best to read this report in conjunction 
with the document Conclusions and Comparisons from the 1.3 Results. 
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O. Special Note About RMI and JBoss 3.0.0 

Most undercover tests are based on the principle that many test clients will make almost 
simultaneous RMI calls to the EJB server. Hie JBoss version tested, however, exhibited 
a particular problem with this type of load: when the number of clients in the burst 
exceeded 100, the test would usually fail with multiple RMI ConnectExceptions. 

Discussion in the JBoss Forums established that this was a known problem expected to be 
fixed in a later minor release, possibly version 3.0.3. However, the policy for 
undercover testing (see section A) is that the first production-capable version for each 
J2EE release is put through the tests. In order to be fair to all vendors, this policy has 
been adhered to, and the following results are based on JBoss version 3.0.0. This is why 
the results show lest cases of up to only 1 00 or sometimes 200 clients. 

The decision to release JBoss 3.0 with such a signilicant problem was made by the JBoss 
community, and it seems reasonable that they should take responsibility for it. It is worth 
noting, however, that as on open-source product, JBoss would have certain migrating 
advantages. Any customers in a grave situation with this problem would have the option 
of looking.at the product source code and possibly patching it themselves. This 
consideration is useful for understanding why it is sometimes difficult to compare JBoss 
on a like-for-likc basis with more commercial alternatives. 
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II. Session Bean Tests 



A. Pooling of Stateless Session Beans 




Fig 1. Results of Stateless Session Bean Pooling Test 

The pooling tests are designed to determine whether an application server pools stateless 
session bean instances, and if possible, identify factors which influence the size of the 
pool. Each of the points on the graph represents a test where a burst of client requests has 
been made on an individual session bean. At the left end arc tests where only a single 
client request has been made; at the right end are tests where 200 client requests have 
been made in a single burst. The Y-axis reveals the number of instances of the session 
bean which were used to handle each load. 

The JBoss 3 results show that a high degree of instance pooling is occurring. For a 
lightweight session bean which does very little processing ("No Cak"), all of the client 
loads can be handled by a single instance. 

The "Heavy Cale" test run shows the same test for a session bean which performs 
significant processing - each bean invocation results in one million square root 
calculations. Here the number of beans required is larger than in the no calculation case, 
but there is still significant pooling occurring. 

So far as they have succeeded, these results are similar to those of other J2EE 1,3 
application servers. Unfortunately it has not been possible to get results up to the 
standard test load of 5000 clients. (For explanation see section LD). 
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B. Stateful Session Beans 



Bl. Instances of Stateful Session Beans 

Because stateful session beans hold state which is specific to a client, their instances are 
not pooled. A pooling test such as that shown in section A (above) on JBoss 3 stateful 
beans shows that the number of bean instances used is always equal to the number of 
clients in the burst; the graph shows y = x. 

In theory, there is some latitude for application server vendors to use an implementation 
which would not always ensure one bean instance for each client; however, all of the 
J2EC 1.3 products tested to date show this result for stateful beans. 



B2. Passivation of Stateful Session Beans 

One area where the products are not all alike is passivation. The potentially large number 
of stateful bean instances can cause a shortage of memory; consequently application 
servers are allowed to passivate instances to disk. However, it is not easy to observe this 
behaviour on certain products, and it is possible their designers have decided not to 
implement passivation. 

Unfortunately, due to the problems of RM1 load testing on JBoss 3.0.0 (see explanation 
in section l.D) it has not been possible to do a meaningful test for passivation in JBoss. 
However, discussions in the JBoss forums suggest that the product does passivate stateful 
beans in practice. 
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C. Stateless and Stateful Session Performance 




5 10 20 50 
Number of Clients fn Burst 



Fig 2. Stateless vs. Stateful Session Performance Comparison 

The performance comparison shows a series of tests which have been timed. The total 
time from when the clients are started until they arc all serviced is measured and then 
divided by the number of clients. This average time provides a measure of how 
efficiently the application server is handling the load. Because undercover testing is not a 
performance benchmark, the Y-axis is expressed in relative units, where *f average tune 
for the first stateless test has been assigned an arbitrary reference value ot 100. 

Two series of teste are shown: one where the undercover session bean is glared to be 
stateless and one where it is declared stateful. It has already been demonstrated that the 
stateful tests will involve the creation of a larger number of bean instances than the 
stateless teste. The key question for this test to answer is whether that difference will 
have a significant impact on performance. 

In both cases there is an encouraging decrease in average time as the number of clients 
becomes significant. Possibly due to the number of bean instances created however the 
stateful teste perform somewhat worse than the stateless ones. Unfortunately it is ^ot 
possible to see if this difference continues into higher loads because of the problems with 
RMI load testing (see section I.D). 
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III. Message-Driven Bean Tests 



Tests confirm that JBoss 3 pools Message Driven Bean (MDB) instances in a manner 
similar to the pooling of stateless session bean instances. For example, a burst of 10 
clients that send messuges to the same JMS topic will be serviced by a single MDB 
instance. 



Unfortunately, tests succeed reliably only when the number of clients in the burst is 20 
lower. Bursts of 50 JMS clients will often fail. Unlike the problems with RM1 load 
testing, this problem under JMS load is quite common among J2EE 1 .3 application 
servers. As with all of them, it is possible that configuration and tuning will help. 



It is also worth remembering that the current undercover MDB implementation opens and 
closes a new connection on behalf of each client publisher; a more typical load scenario 
would be that where a few publishers each publish many messages. This and other 
known problems are discussed in Introduction to Undercover Components. 
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IV. Entity Bean Tests 



A. 



Bean Instances and Primary Keys 
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Fig 4, Entity Bean Contention Tests - 20 Clients in Each Burst 

N.B. Due to RM1 load problems (see section LI)) the standard 1000 clients have not 
been used in this test scries. Instead a load of 20 clients in each burst was used. 

The contention tests reveal what the determining factor is for how many entity bean 
instances ore used. In some products, there is a clear relationship maintained between 
primary keys and bean instances, whereas in others, the bean instances are generic tools 
that are created in response to total client load. Each of the tests shown above involves a 
burst of 20 clients which are contending over a variable number of primary keys. The 
test on the far right shows a case where all 20 clients are targeting the same primary key. 

The results demonstrate that J Boss 3 is maintaining a relationship between bean instances 
and primary keys. In fact, there is a consistent result that the number of bean instances 
created during a given test is exactly the number of distinct primary keys plus the number 
of clients. This suggests that, once in the in the ready state, beans remain there to be re- 
used by those seeking the same primary key. The impact from the number of clients is 
harder to understand. One possibility is that each client is causing its own bean instance 
to be created into the pooled state even though it ought to be possible to re-usc them. 
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S. Commit Options A/B/C 




Fig 5. Commit Option Test - 20 Clients on 5 Primary Keys 

N B Due to RMT load problems (see section I.D) the standard 100 clients against 20 
have not been used in this test series. Instead a load of 20 clients against 5 keys 
was used. 

The commit option test examines a single burst of client activity and all of the callback 
notifications that result. In this case, 20 clients were started and each one followed this 
procedure: 

* Look up the home interlace for the undercover entity bean. 

. Call findByPrimaryKcyO using an integer primary key determined by calculating 
the client number modulo 5. 

• Call the business method of the entity bean. 

The resulting counts of each kind of callback notification reveal which commit option is 
being used by default, and may also spot unusual activity. 

The result above shows that JBoss is, by default, using an Option A commit. This is 
evident since the number of ejbLoad notifications is much lower than the ™mber ot 
business method calls. This indicates that once the server has loaded the state of the bean 
„; s »ni from the database, it assumes that that state is valid from then on and docs not 
refresh it later when the bean is rc-uscd. 

The "standardiboss xml" configuration tile has many switches relating to commit options, 
and B tS -ggefts Option A will not be the default » all enUty bean 
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cases. Since the undercover entity bean is a bean-managed persistence (BMP) bean 
running on a standalone (non-clustered) server, it has defaulted to Option A behaviour. 
However, several other varieties of entity beans, including all of those deployed in a 
cluster, will default to Option B behaviour. 

Incidentally, the unusually high number of newlnstancc and setContext calls is consistent 
with the suggestion (in section A) that JBoss is creating a new entity instance in the 
pooled state to handle each client. It is hard to see why the instances are not being re- 
used. 
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V. General Comments 

A. Documentation, Examples, and Ease-of-Use 

Subjective Grade for Documentation, Examples, and Ea»e-of-Use: F 

(possible grades arc A-F) 

Documentation is a problem for open-source projects, and the JBoss* O^pi Pjg"^. 
that developers working without pay can not be motivated to produce it Therefore JBoss 
has a program of documenlation-for-sale and provides a free Getting Started guide. 
UnfoLately, in the period immediately after the ^^^1^ started 
documentation was available, either for-sale or free. The only obtainable Getting biartea 
guide was for the previous 2.x version, and it was disappointing. 

Even finding what documentation and examples did exist was difficult since they were 
noUncSd in the main product download, and the JBoss web site was confusing for this 
purpose Significant time" was wasted in tracking down mree separate downloads. Once 
downloaded, it became clear that a lot of trial and error would be 'f^ ™' 
experience is simply not comparable to that of downloading one of the commercial 
products which has made a serious effort to help you get started. 



B. Key Points on JBoss 3 

JBoss is different. Its proponents point out several advantages, most importantly that it's 
free Unlike the J2EE Reference Implementation, which is also free, JBoss is licensed tor 
use in production systems. Those with a difficult support problem and ^ju^m* 
can study the server source code. Finally, the undercover tests confirm that there is a real 
EJB container there, since it is capable of executing the tests. 

But the tests reveal real problems too. The RM1 load (section I.D) and extra entity 
^stances (section IV) issues are significant. The effort to execute the tests *o showed a 
problem with true J2F.E compatibility. Undercover tests make use of the application 
dk£ container" facility which is required by J2EE 1.3. This resolves the EJB preference, 
used by the client. Unlike all other J2EE 1.3 products tested, however, JBoss has not 
implemented this facility and it was necessary to change the clients to use JNDI ] aames 
directly. Given that JBoss has not been subjected to the ngors of J2EE compatibility 
testing, ii would be surprising if dus were the only omission. 

Those who are sensitive to issues of strict compatibility or quality assurance in release 
procedures should note the particular disadvantages of JBoss as well as its advantages. 
Those with tight budgets, their own J2EE expertise, and the extra time necessary to 
benefit from the selling points of JBoss, will be glad to have this distinct alternative. 
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C. summary and History of Observations 

dement Conclusion, and Comparison from Ih. U *«"'«. 
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